Skip to content

Add a metastore read replica role for read-only routing#6548

Open
shuheiktgw wants to merge 16 commits into
quickwit-oss:mainfrom
shuheiktgw:metastore-read-replica-role
Open

Add a metastore read replica role for read-only routing#6548
shuheiktgw wants to merge 16 commits into
quickwit-oss:mainfrom
shuheiktgw:metastore-read-replica-role

Conversation

@shuheiktgw

@shuheiktgw shuheiktgw commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR adds support for deploying metastore read replicas as a dedicated metastore_read_replica service and lets searchers opt into using them through searcher.use_metastore_read_replica.

How it works

  1. Users deploy standalone metastore_read_replica nodes with metastore_read_replica_uri pointing to a PostgreSQL read replica. If searcher.use_metastore_read_replica is true, searchers use metastore_read_replica instead of the primary metastore.
  2. Searchers use a dedicated read-only metastore client, MetastoreReadServiceClient, which exposes only the read-only subset of metastore RPCs to prevent write methods from being used.

Limitations / Future Improvements

  • metastore_read_replica needs to run as a standalone service to keep the implementation simple. It shares the same gRPC service as metastore, so it cannot run alongside it, and running it separately also simplifies the metadata service connection handling.
  • Currently, searchers configured to use metastore read replicas still need access to the primary metastore. Quickwit treats every node as an API server, and some API requests need to write to the metastore. Therefore, even read-replica-backed searchers validate connectivity to the primary metastore and include it in their health checks. This means that if the primary metastore is unavailable, those searchers are marked unhealthy as well. We can improve this later.
  • There are many more places where we could use the read replica metastore, but the primary focus of this PR is the searcher. We can expand support further later.

@shuheiktgw shuheiktgw force-pushed the metastore-read-replica-role branch 2 times, most recently from c0ae4dd to d14a422 Compare June 25, 2026 16:13
Comment thread config/quickwit.yaml
# # DataFusion when enabled, to nodes running the `metastore_read_replica`
# # service. Searchers require at least one `metastore_read_replica` node at
# # startup and do not fall back to the primary metastore.
# use_metastore_read_replica: false

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason we need this flag in addition to metastore_read_replica_uri is that users can set metastore_read_replica_uri via the QW_METASTORE_READ_REPLICA_URI environment variable. Without this flag, searchers would need access to that same environment variable even though they don’t need the secret itself.

Introduce the `metastore_read_replica` service role and the
`metastore_read_replica_uri` node config option: the foundation for
routing read-only metastore traffic to a PostgreSQL read replica.

- Add `QuickwitService::MetastoreReadReplica`, parsed from
  `metastore_read_replica` / `metastore-read-replica`.
- Split `QuickwitService::default_services()` from `supported_services()`
  so the new role is opt-in and never enabled implicitly on all-in-one
  nodes.
- Add the optional `metastore_read_replica_uri` field (env
  `QW_METASTORE_READ_REPLICA_URI`), redacted alongside `metastore_uri`.
- Validate that the role requires a PostgreSQL `metastore_read_replica_uri`
  and cannot be co-located with the `metastore` role.
Add the resolution plumbing for connecting to a PostgreSQL read replica
over a read-only connection.

- Add `MetastoreFactoryOptions { read_only }` and thread it through
  `MetastoreFactory::resolve`.
- Add `MetastoreResolver::resolve_read_only`, which rejects any non-
  PostgreSQL backend.
- Key the PostgreSQL factory cache on `(uri, options)` so the read-write
  and read-only clients get distinct connection pools.
- Add `PostgresqlMetastore::new_read_only`: a read-only connection pool
  with migrations skipped (the replica is migrated by the primary).
Resolve and expose a metastore gRPC server when the
`metastore_read_replica` role is enabled, backed by a read-only
connection to `metastore_read_replica_uri`.

- The read replica server reuses the metrics + load-shed layers but omits
  the control-plane event layers, which only wrap write RPCs.
- A read-replica node is exempted from the control-plane connectivity
  wait, like a primary metastore node, so dedicated replica pods start
  independently.
- Extract `metastore_max_in_flight_requests` shared by both roles.
Add `ReadReplicaRoutingMetastore`, a `MetastoreService` wrapper that
routes the stale-tolerant reads issued by the search and analytics paths
(`index_metadata`, `indexes_metadata`, `list_indexes_metadata`,
`list_splits`, `list_metrics_splits`, `list_sketch_splits`) to read
replica nodes when any are connected, and everything else (writes and
non-hot-path reads) to the primary.

- Routing is decided per request from the read replica balance channel's
  live connection set, so it degrades to the primary when no replica is
  deployed. The check is a synchronous `watch` read with no borrow held
  across an await.
- Wire it into the searcher service and the DataFusion session builder, so
  all search (REST, Elasticsearch, gRPC) and metrics analytics benefit,
  while REST admin handlers keep read-your-writes against the primary.
- Document `metastore_read_replica_uri` in the node config reference and
  the example `quickwit.yaml`.
- Add a PostgreSQL-gated test that resolves a read-only metastore against
  a real database and verifies it serves read RPCs.
Replace the full `MetastoreService` implementation on
`ReadReplicaRoutingMetastore` (which forced ~45 delegating methods, most
of them writes) with a narrow, read-only `MetastoreReadService` trait —
the read-only subset of the metastore RPCs (`index_metadata`,
`list_indexes_metadata`, `list_splits`, `list_metrics_splits`,
`list_sketch_splits`).

- `MetastoreServiceClient` implements `MetastoreReadService`;
  `MetastoreReadServiceClient = Arc<dyn MetastoreReadService>`.
- `ReadReplicaRoutingMetastore` now implements only the 5-method trait, so
  writes are excluded at the type level rather than delegated.
- The search and DataFusion read paths take `MetastoreReadServiceClient` /
  `&dyn MetastoreReadService`; `list_parquet_splits_*` take
  `&dyn MetastoreReadService`. `single_node_search` keeps its concrete
  `MetastoreServiceClient` parameter and adapts internally.

Addresses review feedback that the wrapper should expose a Go-style
read-only interface instead of reimplementing the whole service.
@shuheiktgw shuheiktgw force-pushed the metastore-read-replica-role branch from 1bf29fc to 09c32ad Compare June 26, 2026 15:16
@shuheiktgw shuheiktgw marked this pull request as ready for review June 26, 2026 15:28
@shuheiktgw shuheiktgw requested review from a team as code owners June 26, 2026 15:28

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09c32ada6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread quickwit/quickwit-serve/src/lib.rs
Comment thread quickwit/quickwit-serve/src/lib.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant